<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML
><HEAD
><TITLE
>DataparkSearch performance issues</TITLE
><META
NAME="GENERATOR"
CONTENT="Modular DocBook HTML Stylesheet Version 1.79"><LINK
REL="HOME"
TITLE="DataparkSearch Engine 4.52"
HREF="index.en.html"><LINK
REL="UP"
TITLE="Storing data"
HREF="dpsearch-howstore.en.html"><LINK
REL="PREVIOUS"
TITLE="Cache mode storage"
HREF="dpsearch-cachemode.en.html"><LINK
REL="NEXT"
TITLE="SearchD support"
HREF="dpsearch-searchd.en.html"><LINK
REL="STYLESHEET"
TYPE="text/css"
HREF="datapark.css"><META
NAME="Description"
CONTENT="DataparkSearch - Full Featured Web site Open Source Search Engine Software over the Internet and Intranet Web Sites Based on SQL Database. It is a Free search software covered by GNU license."><META
NAME="Keywords"
CONTENT="shareware, freeware, download, internet, unix, utilities, search engine, text retrieval, knowledge retrieval, text search, information retrieval, database search, mining, intranet, webserver, index, spider, filesearch, meta, free, open source, full-text, udmsearch, website, find, opensource, search, searching, software, udmsearch, engine, indexing, system, web, ftp, http, cgi, php, SQL, MySQL, database, php3, FreeBSD, Linux, Unix, DataparkSearch, MacOS X, Mac OS X, Windows, 2000, NT, 95, 98, GNU, GPL, url, grabbing"></HEAD
><BODY
CLASS="SECT1"
BGCOLOR="#FFFFFF"
TEXT="#000000"
LINK="#0000C4"
VLINK="#1200B2"
ALINK="#C40000"
><!--#include virtual="body-before.html"--><DIV
CLASS="NAVHEADER"
><TABLE
SUMMARY="Header navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TH
COLSPAN="3"
ALIGN="center"
>DataparkSearch Engine 4.52: Reference manual</TH
></TR
><TR
><TD
WIDTH="10%"
ALIGN="left"
VALIGN="bottom"
><A
HREF="dpsearch-cachemode.en.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="80%"
ALIGN="center"
VALIGN="bottom"
>Chapter 5. Storing data</TD
><TD
WIDTH="10%"
ALIGN="right"
VALIGN="bottom"
><A
HREF="dpsearch-searchd.en.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
></TABLE
><HR
ALIGN="LEFT"
WIDTH="100%"></DIV
><DIV
CLASS="SECT1"
><H1
CLASS="SECT1"
><A
NAME="PERF"
>5.3. <SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
> performance issues</A
></H1
><A
NAME="AEN2926"
></A
><P
>The cache mode is the fastest <SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
>'s storage mode. Use it if
you need maximal search speed.</P
><P
><A
NAME="AEN2933"
></A
>
If your <TT
CLASS="FILENAME"
>/var</TT
> directory isn't changed since the indexing has been finished, you may disable file locking using 
<B
CLASS="COMMAND"
>"ColdVar yes"</B
> command placed in <TT
CLASS="FILENAME"
>search.htm</TT
> (or in <TT
CLASS="FILENAME"
>searchd.conf</TT
>,
if <B
CLASS="COMMAND"
>searchd</B
> is used). This allow you to save some time on file locking.</P
><P
>Using <KBD
CLASS="USERINPUT"
>UseCRC32URLId yes</KBD
> command (see <A
HREF="dpsearch-indexcmd.en.html#USECRC32URLID_CMD"
>Section 3.10.12</A
>&#62;) allow to speed up indexing, but
small number of collisions is possible, especially on large database. </P
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="SEARCHD-REC"
>5.3.1. <B
CLASS="COMMAND"
>searchd</B
> usage recommendation</A
></H2
><P
>If you plan use ispell data, synonym or stopword lists, it's recommended setup
the <B
CLASS="COMMAND"
>searchd</B
> daemon for speed-up searches (See <A
HREF="dpsearch-searchd.en.html"
>Section 5.4</A
>&#62;). 
<B
CLASS="COMMAND"
>searchd</B
> daemon preload all these data and lists
and holds them in memory. This reduce average search query execution time.</P
><P
>Also, <B
CLASS="COMMAND"
>searchd</B
> can preload url info data (20 bytes per URL indexed) and 
cache mode limits (4 or 8 bytes per URL depend on limit type). This allow reduce average search time.</P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="MFS-REC"
>5.3.2. Memory based filesystem (mfs) usage recommendation</A
></H2
><P
>If you use cache storage mode and you have enough RAM on your PC, 
you may place <TT
CLASS="FILENAME"
>/usr/local/dpsearch/var</TT
>
directory on memory based filesystem (mfs). This allow speedup both indexing and searching.</P
><P
>If you haven't enough RAM to fit  <TT
CLASS="FILENAME"
>/usr/local/dpsearch/var</TT
>, 
you may place on memory filesystem any of <TT
CLASS="FILENAME"
>/usr/local/dpsearch/var/tree</TT
>,
<TT
CLASS="FILENAME"
>/usr/local/dpsearch/var/url</TT
> or <TT
CLASS="FILENAME"
>/usr/local/dpsearch/var/store</TT
>
directories as well.</P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="URLINFO-CMD"
>5.3.3. <B
CLASS="COMMAND"
>URLInfoSQL</B
> command</A
></H2
><A
NAME="AEN2965"
></A
><P
>For dbmode cache, you may use <B
CLASS="COMMAND"
>URLInfoSQL no</B
> command to disable storing URL Info into SQL database.
But using this command, you'll be unable to use limits by language and by Content-Type.</P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="MARKFORINDEX-CMD"
>5.3.4. <B
CLASS="COMMAND"
>MarkForIndex</B
> command</A
></H2
><A
NAME="AEN2973"
></A
><P
>By default, <SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
> are marking all URLs selected for indexing as indexed for 4 hours.
This prevent possible simultaneous indexing of the same URL by different indexer instance running. But for huge installation this feature
can take some time for processing. You may switch off this markage using <B
CLASS="COMMAND"
>"MarkForIndex no"</B
> in your 
<TT
CLASS="FILENAME"
>indexer.conf</TT
> file.</P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="CHECKINSERTSQL-CMD"
>5.3.5. <B
CLASS="COMMAND"
>CheckInsertSQL</B
> command</A
></H2
><A
NAME="AEN2983"
></A
><P
>By default, <SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
> trying to insert data into SQL database regardless it's already present there.
On some systems this raise some error loggings. To avoid such errors, 
you may enable additional checks, is the inserting data new, by specifying
<B
CLASS="COMMAND"
>CheckInsertSQL yes</B
> command in your <TT
CLASS="FILENAME"
>indexer.conf</TT
>.</P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="PERF-MYSQL"
>5.3.6. MySQL performance</A
></H2
><P
>MySQL users may declare <SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
> tables with
<CODE
CLASS="OPTION"
>DELAY_KEY_WRITE=1</CODE
> option. This will make the updating
of indexes faster, as these are not logged to disk until the file is
closed. <CODE
CLASS="OPTION"
>DELAY_KEY_WRITE</CODE
> excludes updating indexes on
disk at all. </P
><P
>With it indexes are processed only in memory and
written onto disk as last resort, command <B
CLASS="COMMAND"
>FLUSH TABLES</B
> or mysqld shutdown. This can take even minutes and
impatient user can <TT
CLASS="LITERAL"
>kill -9 mysql server</TT
> and break
index files with this.  Another downside is that you should run
<TT
CLASS="LITERAL"
>myisamchk</TT
> on these tables before you start mysqld
to ensure that they are okay if something killed mysqld in the
middle.</P
><P
>Because of it we didn't include this table
option into default tables structure. However as the key information
can always be generated from the data, you should not lose anything by
using <CODE
CLASS="OPTION"
>DELAY_KEY_WRITE</CODE
>. So, use this option for your
own risk.</P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="PERF-OPTIMIZATION"
>5.3.7. Post-indexing optimization</A
></H2
><P
>This article was supplied by Randy Winch <CODE
CLASS="EMAIL"
>&#60;<A
HREF="mailto:gumby@cafes.net"
>gumby@cafes.net</A
>&#62;</CODE
>
		</P
><P
>I have some performance numbers that some of you
might find interesting. I'm using RH 6.2 with the 2.2.14-6.1.1 kernel
update (allows files larger than 2 gig) and mysql 2.23.18-alpha.  I
have just indexed most of our site using mnoGoSearch 3.0.18:
		<PRE
CLASS="PROGRAMLISTING"
>&#13;          mnoGoSearch statistics

    Status    Expired      Total
   -----------------------------
       200     821178    2052579 OK
       301        797      29891 Moved Permanently
       302          3          3 Moved Temporarily
       304          0          7 Not Modified
       400          0         99 Bad Request
       403          0          7 Forbidden
       404      30690     100115 Not found
       500          0          1 Internal Server Error
       503          0          1 Service Unavailable
   -----------------------------
     Total     852668    2182703&#13;</PRE
></P
><P
>I optimize the data by dumping it into a file
using <CODE
CLASS="OPTION"
>SELECT * INTO OUTFILE</CODE
>, sort it using the system
sort routine into word (CRC) order and then reloading it into the
database using the procedure described in the 
<A
HREF="http://www.mysql.com/documentation/mysql/commented/manual.php?section=Insert_speed"
TARGET="_top"
>mysql online manual</A
>.</P
><P
>The performance is wonderful. My favorite test
is searching for "John Smith". The optimized database version takes
about 13 seconds. The raw version takes about 73 seconds.</P
><PRE
CLASS="PROGRAMLISTING"
>Search results: john : 620241 smith : 177096
Displaying documents 1-20 of total 128656 found</PRE
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="ARES"
>5.3.8. Asynchronous resolver library</A
></H2
><P
>Using <A
HREF="ftp://athena-dist.mit.edu/pub/ATHENA/ares/"
TARGET="_top"
>ares</A
>, 
an asynchronous resolver library (dns/ares in FreeBSD ports collection), allow to perform DNS queries without blocking for every
indexing thread. Please note, this also increase the number of concurrent queries to your DNS server.</P
></DIV
></DIV
><DIV
CLASS="NAVFOOTER"
><HR
ALIGN="LEFT"
WIDTH="100%"><TABLE
SUMMARY="Footer navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
><A
HREF="dpsearch-cachemode.en.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="index.en.html"
ACCESSKEY="H"
>Home</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
><A
HREF="dpsearch-searchd.en.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
>Cache mode storage</TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="dpsearch-howstore.en.html"
ACCESSKEY="U"
>Up</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
>SearchD support</TD
></TR
></TABLE
></DIV
><!--#include virtual="body-after.html"--></BODY
></HTML
>